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Visual mismatch negativity (vMMN), a component in event-related potentials (ERPs), can 
be elicited when rarely presented "deviant" facial expressions violate regularity formed 
by repeated "standard" faces. vMMN is observed as differential ERPs elicited between 
the deviant and standard faces. It is not clear, however, whether differential ERPs to 
rare emotional faces interspersed with repeated neutral ones reflect true vMMN (i.e., 
detection of regularity violation) or merely encoding of the emotional content in the 
faces. Furthermore, a face-sensitive N170 response, which reflects structural encoding of 
facial features, can be modulated by emotional expressions. Owing to its similar latency 
and scalp topography with vMMN, these two components are difficult to separate. We 
recorded ERPs to neutral, fearful, and happy faces in two different stimulus presentation 
conditions in adult humans. For the oddball condition group, frequently presented neutral 
expressions (p = 0.8) were rarely replaced by happy or fearful expressions (p = 0.1), 
whereas for the equiprobable condition group, fearful, happy, and neutral expressions 
were presented with equal probability (p = 0.33). Independent component analysis (ICA) 
revealed two prominent components in both stimulus conditions in the relevant latency 
range and scalp location. A component peaking at 130 ms post stimulus showed a 
difference in scalp topography between the oddball (bilateral) and the equiprobable 
(right-dominant) conditions. The other component, peaking at 170 ms post stimulus, 
showed no difference between the conditions. The bilateral component at the 130-ms 
latency in the oddball condition conforms to vMMN. Moreover, it was distinct from N170 
which was modulated by the emotional expression only. The present results suggest 
that future studies on vMMN to facial expressions should take into account possible 
confounding effects caused by the differential processing of the emotional expressions 
as such. 

Keywords: equiprobable condition, facial expressions, independent component analysis, oddball condition, visual 
mismatch negativity 



INTRODUCTION 

Other people's facial expressions convey socially important infor- 
mation about other individuals' emotions and social intentions 
(Keltner et al, 2003). Therefore, it is not surprising that facial 
expressions are, among other biologically and socially significant 
information, processed automatically and rapidly in the brain 
(e.g., Adolphs, 2002; Palermo and Rhodes, 2007). 

Because of its good time resolution, measurement of event- 
related potentials (ERPs) has been widely used in studies inves- 
tigating the early stages of facial information processing. An 
ERP component called visual mismatch negativity (vMMN; 
visual counterpart of mismatch negativity, defined originally in 
the auditory modality, Naatanen et al., 1978; for a review see 
Naatanen et al, 2010) is a feasible method to study automatic 
encoding of several types of visual stimuli including faces. vMMN 
is elicited to rare stimuli ("deviant") interspersed with repeated 
("standard") stimuli and observed as a differential ERP response 



between these two. vMMN can be observed in conditions where 
the participants are instructed to ignore the visual stimuli eliciting 
the vMMN and attend to other visual stimuli (e.g., Stefanics et al., 
2012) or auditory stimuli (e.g., Astikainen and Hietanen, 2009). 
In addition to changes in low-level visual features, such as orien- 
tation of a bar (e.g., Astikainen et al., 2008) or color (e.g., Czigler 
et al, 2002), it has also been associated with changes in complex 
visual features, including human hands (Stefanics and Czigler, 
2012) and facial expressions (Susac et al., 2004; Zhao and Li, 2006; 
Astikainen and Hietanen, 2009; Chang et al., 2010; Kimura et al., 
2011; Li et al, 2012; Stefanics et al, 2012). 

vMMN is considered to reflect a process of detecting a mis- 
match between the representation of the repeated standard stim- 
ulus in transient memory and the current sensory input (Czigler 
et al, 2002; Astikainen et al, 2008; Kimura et al., 2009) simi- 
larly to auditory MMN (for the trace -mismatch explanation of 
MMN, see Naatanen, 1990). The standard stimuli can also be 
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physically variant, but if they form sequential regularity, deviant 
stimuli violating this regularity elicit vMMN (Astikainen and 
Hietanen, 2009; Kimura et al, 2010, 2011; Stefanics et al, 2011, 
2012; for a review see Kimura, 2012). For example, serially pre- 
sented pictures of faces can be of different identities, but a 
vMMN is elicited if, say, rare fearful faces are interspersed among 
emotionally neutral faces, suggesting that a representation of a 
"neutral face" can be abstracted among several low-level features 
(Astikainen and Hietanen, 2009). Along the same lines, vMMN 
elicitation has recently been linked to the predictive coding the- 
ories (Friston, 2005), which postulate a predictive error between 
the neural model based on the representations of visual objects 
in memory and the actual perceptual input (Winkler and Czigler, 
2012). 

vMMN to facial expressions has been reported at different 
latency ranges, starting from 70 up to 360 ms post-stimulus 
(Susac et al., 2004; Zhao and Li, 2006; Astikainen and Hietanen, 
2009; Chang et al, 2010; Kimura et al, 2011; Li et al, 2012; 
Stefanics et al., 2012), and sometimes multiple responses with 
different latencies have been reported (Astikainen and Hietanen, 
2009; Chang et al, 2010; Li et al, 2012; Stefanics et al, 2012). 
Nevertheless, a consistent finding has been a vMMN observed 
around the same latency (~ 130-200 ms after stimulus onset) and 
in the same scalp location (parieto-occipital region) with the 
well-known face-sensitive N170 response (Zhao and Li, 2006; 
Astikainen and Hietanen, 2009; Chang et al, 2010; Stefanics et al, 
2012). The N170 was originally associated with the structural 
encoding of faces (Bentin et al., 1996), but several studies have 
shown its sensitivity to emotional expressions (Batty and Taylor, 
2003; Eger et al, 2003; Caharel et al, 2005; Williams et al, 2006; 
Blau et al., 2007; Leppanen et al., 2007; Schyns et al., 2007; Japee 
et al., 2009; Vlamings et al., 2009; Wronka and Walentowska, 
2011, for the studies showing no emotional modulation of N 170, 
see Eimer and Holmes, 2002; Eimer et al., 2003; Holmes et al., 
2003, 2005; Ashley et al, 2004; Santesso et al, 2008). Because the 
N170 can be modulated by emotional expressions, and because 
its latency and scalp topography can resemble the vMMN to 
facial expressions, differentiating these two components is diffi- 
cult. This is especially true in vMMN studies in which emotional 
faces have been used as deviant faces among neutral standard 
faces (Zhao and Li, 2006; Astikainen and Hietanen, 2009; Chang 
et al., 2010; Li et al., 2012) since the differential response could 
result from enhanced N170 responses to emotional vs. neutral 
faces. 

Moreover, assuming that vMMN to emotional facial expres- 
sions could be separated from N170, there might be an additional 
confounding factor to consider. Namely, it is unclear whether 
such a differential response (seemingly similar to vMMN) reflects 
a true mismatch response that is, a response indicating regular- 
ity violation. The other possibility is that the differential response 
reflects, solely or in part, varying levels of sensitivity to different 
facial emotions. A few recent studies have elucidated this issue. In 
the study by Stefanics et al. (2012), regularity violations involved 
rare changes in emotional expressions (infrequent fearful face 
among happy faces and vice versa) of constantly changing facial 
identities. A rarely presented facial expression elicited differen- 
tial ERPs relative to the same emotion when it was presented as 



a frequent one (i.e., happy standard vs. happy deviant face, fear- 
ful standard vs. fearful deviant face) at 70-120 ms latency for the 
fearful faces, and at 170-360 ms latency covering N170 and P2 
components for both the fearful and happy faces. In the study by 
Kimura et al. (2011), an immediate repetition of an emotional 
expression was presented as a deviant stimulus violating the pat- 
tern of constantly changing (fearful and happy) emotions while 
the participants were attending to faces wearing eyeglasses. This 
stimulus condition elicited responses associated to the regularity 
violation at relatively long latencies: ~280 ms after the onset of the 
fearful faces and 350 ms after the onset of the happy faces. In both 
of these studies, the experimental paradigms allowed the analysis 
of vMMN by comparing the ERPs elicited by two identical pic- 
tures (e.g., fearful faces presented as a standard and as a deviant 
stimulus). Since differential ERPs, i.e., vMMN, to these physi- 
cally identical pictures were found, confounding by emotional 
processing as such can be ruled out. 

The existing findings of vMMN as an index of regularity vio- 
lation in facial expression processing are, however, only from 
experiments which applied happy and fearful faces in the stim- 
ulus series, i.e., all the expression used in the experiments were 
emotional expressions (Kimura et al., 201 1; Stefanics et al., 2012). 
vMMN as an index of regularity violation and its possible con- 
founding by emotional expression encoding remains an open 
question in the case of expressive vs. neutral faces. In our previous 
study (Astikainen and Hietanen, 2009), neutral standard faces of 
constantly changing identities were presented, and the regularity 
violations were rarely presented fearful or happy faces. A differ- 
ential response was elicited by the rare expressions at 150-180 ms 
latency, but this study left open the question of functional inde- 
pendency between N170 and vMMN as well as the question of the 
emotional confounding of the vMMN response. The same holds 
true for a study in which regularity of "neutral expression" was 
violated by happy and sad faces while changing "identities" (i.e., 
low-level visual features) of schematic faces were used (Chang 
et al.,2010). 

In order to reveal the process underlying the differential 
responses to deviant emotional vs. neutral standard faces (reg- 
ularity violation vs. encoding of emotional expression) and its 
independency from (emotion-modulated) N170, we recorded 
ERPs in two different conditions presented to two groups of 
adult humans. For one group, happy, fearful, and neutral faces 
were presented in random order and with an equal probabil- 
ity (p = 0.33 for each; equip rob able condition). For the other 
group, fearful and happy faces were rarely (p = 0.1 for both) 
and pseudo- randomly (at least two neutral faces were presented 
between the emotional faces) interspersed with the neutral ones 
(oddball condition). In both conditions, the facial identities 
changed from trial to trial requiring abstraction of the regularity 
in the facial expressions from among several low-level features. 
Independent component analysis (ICA) was applied to the data. 
ICA functions as a spatial filter for the ERP data, and the peak 
amplitudes of the components projected to the sensor space can 
be further used in statistical analysis. 

We expected to find two separate components in the 
oddball condition: emotion modulated N170 and vMMN. 
In the equiprobable condition, we expected to observe only 
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emotion-modulated N170. It was also possible that no differential 
response would be found in the equip robable condition if N 170 
was not sensitive to the emotional facial expressions in the present 
stimulus presentation condition. 

METHODS 
PARTICIPANTS 

Twenty native Finnish- speaking volunteers participated in the 
study. For half of the participants, the stimuli were presented in 
the oddball condition whereas the other half viewed the stimuli in 
the equiprobable condition (see below). Both groups comprised 
two male and eight female participants. In the "oddball" group, 
the age range was 19-35 years and mean age 23.9 years (median 
23.5). In the "equiprobable" group, the age range was 20-42 years 
with a mean age of 24.6 years (median 21.5). All the partici- 
pants were right-handed and had self- reported normal hearing 
and vision (corrected with eyeglasses if necessary), and no diag- 
nosed neurological or psychiatric disorders. A written informed 
consent was obtained from the participants before their partici- 
pation. The experiment was undertaken in accordance with the 
Declaration of Helsinki. The ethical committee of the University 
of Jyvaskyla approved the research protocol. 

PROCEDURE 

During the experiment, the participants sat in a comfortable chair 
in a dimly-lit room. The participants were instructed to attend to 
a recording of a radio play. The play was presented via a loud- 
speaker placed at about 50 cm above the participant's head where 
the volume of the recording equaled that of a normal speaking 
voice. Visual stimuli were presented on a computer screen (Eizo 
Flexscan, 17 inch CRT display) approximately one meter away 
from the participant. The participants were monitored during the 
recordings via a video camera positioned on top of the screen. The 
participants were asked to fix their gaze at a cross in the middle 
of a screen. The participants were informed that the visual stim- 
uli would be faces and they were instructed to concentrate on the 
radio play and pay no attention to the faces. 

STIMULI 

The visual stimuli were pictures of faces of four different models 
(male actors PE and J J, female actors MF and NR) from Pictures 
of Facial Affect (Ekman and Friesen, 1976). Pictures of a neu- 
tral, fearful and happy expression from each model were used. 
The stimulus presentation was controlled with E-Prime software 
(Psychology Software Tools, Inc., Sharpsburg, MD, USA). 

The pictures of faces, occupying an area of 4 x 5° of visual 
angle were presented at fixation for 200 ms. The stimulus-onset 
asynchrony (SOA) was 700 ms. The faces were presented in two 
different conditions (a between-subjects variable). In a modified 
oddball condition, two different deviant stimulus types, fearful 
faces and happy faces, were infrequently interspersed between fre- 
quently presented neutral standard faces. Standards and deviants 
were presented pseudo- randomly with the restriction that no less 
than two standards would occur between consecutive deviants. 
Among the 1600 stimuli were 160 happy faces (p = 0.1) and 
160 fearful faces (p = 0.1). In the equiprobable condition, the 
stimulus presentation was otherwise the same, except that all 



the expressions were presented pseudo-randomly (there were no 
immediate repetitions of stimuli from the same emotion cat- 
egory), and with equal probability (p = 0.33). The number of 
stimuli in each emotion category was the same as the number 
of deviants in the oddball condition that is, 160 stimuli in each 
emotion category were presented. In both stimulus presentation 
conditions the facial identity in the pictures changed from trial to 
trial. 

ELECTROENCEPHALOGRAPHY RECORDING 

Electroencephalogram was recorded with Brain Vision Recorder 
software (Brain Products GmbH, Munich, Germany) at Fz, F3, 
F4, Cz, C3, C4, Pz, P3, P4, P7, P8, Oz, Ol, and 02 according to the 
international 10-20 system. An average reference was applied. Eye 
movements and blinks were measured from bipolar electrodes, 
one placed above the left eye and the other lateral to the right 
orbit. The signals from the electrodes were amplified, sampled at 
a rate of 1000 Hz, digitally band pass filtered from 0.05 to 100 Hz, 
and stored on a computer disk. 

DATA ANALYSIS 

The signals from the electrodes were filtered (Butterworth 
zero phase filter: 0.1-30 Hz, 24dB/octave roll-off) and 700- 
ms stimulus-locked segments were extracted (from —100 
to +600 ms). Segments with signal amplitudes beyond the range 
between —100 and 100 |xV in any recording channel, includ- 
ing the EOG channel, were omitted from further analysis. The 
segments were corrected by their baseline values (mean ampli- 
tude during the 100-ms pre-stimulus period). In the equiprobable 
condition, all the responses left after the artifact rejection were 
averaged for each participant. On average, 136 trials for the fear- 
ful (min = 107, max = 155, median = 141), 135 trials for the 
happy (min = 96, max = 154, median = 142), and 135 trials 
for the neutral expression (min = 105, max = 155, median = 
139) were available. In the oddball condition, only responses to 
standards immediately preceding the deviants were averaged. This 
procedure allows the same number of segments, and thus a sim- 
ilar signal-to-noise ratio, for both standards and deviants. On 
average, the number of analyzed trials for the fearful and happy 
deviants and the neutral standards immediately preceding them 
was 138 (fear trials: min = 85, max = 160, median = 149; happy 
trials: min = 78, max = 158, median = 153). Figure 1 depicts 
the ERPs to the happy, fearful, and neutral faces in the oddball 
condition and Figure 2 to those in the equiprobable condition. 

Next, differential ERPs (expressive minus neutral face 
responses) were calculated separately for the fearful and happy 
faces. By this way, the brain activities common to the emotional 
and neutral faces were removed. The differential ERPs were pro- 
cessed by an approach including wavelet filter and ICA. The 
approach and the benefits of it has been thoroughly described 
by Cong et al. (2011a,b, 2012). In this approach, ICA is applied 
to the averaged ERPs (see also Makeig et al., 1997; Vigario and 
Oja, 2008; Kalyakin et al, 2008, 2009; Cong et al, 2011a,b). 
This is different from the commonly used application of ICA 
on the concatenated single-trial EEG data (for a N170 study, see 
e.g., Desjardins and Segalowitz, 2013). Briefly, the method was 
as follows. Wavelet filter was performed on the difference wave. 
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FIGURE 1 | Grand-averaged ERPs in the oddball condition. Raw ERPs to 
happy and fearful deviant faces and to the neutral standard faces 
immediately preceding them. Stimulus onset at time 0. 



EQUIPROBABLE CONDITION 



Happy 



FIGURE 2 | Grand -averaged ERPs in the equiprobable condition. Raw 

ERPs to happy, fearful, and neutral faces. Stimulus onset at time 0. 



Ten levels were set to decompose the signal through the reversal 
biorthogonal wavelet with the order of 6.8, and coefficients at lev- 
els 5-8 were chosen for the reconstruction. The wavelet filter was 
selected so that it could be assumed to remove sensor noise and 
frequencies irrelevant for the studied components (Cong et al., 
2011b, 2012). This has been found to be advantageous in the 



DIFFERENCE ERPs 

Oddball condition 



Equiprobable condition 

Oz P8 



+2 [lV 



- 100 0 600 ms 



Happy 



FIGURE 3 | Grand-averaged differential ERPs (emotional minus neutral 
face). Raw ERPs. 



following ICA decomposition (Cong et al, 2011a,b). Figure 3 
shows the wavelet- filtered differential responses in both stimulus 
presentation conditions. 

The filtered and averaged differential ERPs (responses to fear- 
ful and happy faces minus responses to neutral standard faces) 
were next decomposed by ICASSO software for ICA (Himberg 
et al., 2004). The unmixing matrix was randomly initialized 
100 times, and Fasti CA with the hyperbolic tangent function 
(Hyvarinen, 1999) was run 100 times for each setting to extract 
14 components each time. The 1400 components obtained in the 
100 runs were then clustered into 14 groups using agglomerative 
hierarchical clustering with the average-linkage criterion. Finally, 
the centroid of each cluster was sought and was regarded as one 
component by ICASSO (Himberg et al, 2004). The stability of 
the ICA decomposition was satisfactory: the mean of the index of 
quality (Iq) of the 560 ICA components (2 by 2 by 10 by 14) was 
0.90 (standard deviation = 0.10, min = 0.45, max = 0.99, and 
median = 0.94). This index is for the interpretation of the stabil- 
ity of decomposition for each ICA component. If the Iq is close 
to "1," multiple runs of ICA decomposition give similar results, 
which means that ICA composition is stable. Otherwise, if the 
Iq is close to "0," multiple runs of ICA decomposition give very 
different results. 

After the estimation of the independent components by 
ICASSO, the desired components were chosen based on the peak 
latency between 100 and 200 ms and subsequent evaluation of the 
component's scalp topography (posterior negativity) when the 
component was projected back to the electrodes. This projection 
was to correct the inherent polarity and variance indeterminacy of 
ICA (Makeig et al., 1999). It also allows performing conventional 
statistical analysis on peak amplitudes to reveal the experimen- 
tal effects. Two components with peak latencies of ~130 and 
170 ms after stimulus onset were found for each participant in 
both conditions. 

The peak amplitudes of the electrode -field projection of ICA 
components were submitted to repeated measures multivariate 
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FIGURE 4 | Back-projected 130-ms components for each participant. 

Differential response waveforms (emotional minus neutral) at P7 and P8 
electrodes are drawn separately for fearful and happy faces and for oddball 
condition and equiprobable condition groups. 



analysis of variance (MANOVA) with within- subjects factors of 
Expression (fearful vs. happy) and Electrode (Fz, F3, F4, Pz, P7, 
P8, Oz), and with a between- subjects factor Condition (odd- 
ball condition vs. equiprobable condition). Channel selection was 
based on visual inspection of the grand averaged scalp topography 
maps and previous findings for N170 (e.g., Blau et al, 2007) and 
vMMN to facial emotions (e.g., Astikainen and Hietanen, 2009; 
Stefanics et al., 2012). P-values smaller than 0.05 were consid- 
ered as significant. T-tests were two-tailed, and their test-values 
are reported whenever the p-value is smaller than 0.055. Partial 
eta squared (rip) presents effect size estimates for MANOVA. 

RESULTS 

Figures 1, 2 show the raw grand- averaged ERPs for the oddball 
and equiprobable groups. In Figure 3, the grand- averaged differ- 
ential responses (emotional minus neutral) for both conditions 
are presented. In these figures, differential responses to both emo- 
tional expressions and in both stimulus presentation conditions 
can be observed in the lateral parietal and occipital electrodes in 
the latency range under inspection (100-200 ms post stimulus). 

The ICA decomposition showed two separate components at 
the relevant latency range for both the oddball and equiproba- 
ble condition. The earlier, henceforth 130-ms component, had a 
mean latency of 134 ms, and the later, henceforth 170-ms com- 
ponent, had a mean latency of 165 ms in the posterior electrode 
sites. Figures 4, 5 illustrate the back-projected components in 
individual participants as waveforms at electrodes P7 and P8. 

130-MS COMPONENT 

Figure 6 shows the scalp potential maps for the 130-ms compo- 
nent back-projected to the electrodes in the oddball and in the 
equiprobable conditions. 

A MANOVA revealed a significant main effect of Electrode, 
F (6> 13) = 29.4, p < 0.0001, r\p = 0.931, reflecting the positive 
polarity of the differential response in the frontal electrode sites 
and negative polarity in the posterior electrodes. Importantly, 
an Electrode x Condition interaction was found, F^, 13) = 5.00, 
p = 0.007, T]p = 0.698, indicating a non-homogeneous scalp 
distribution in ERP amplitudes between the two conditions. 
The other main effects or other interaction effects were non- 
significant [Expression: F(i j is) = 0.45, p = 0.834, r\p = 0.003; 
Expression x Condition: i 8 ) = 0.39, p = 0.846, vf p = 0.002; 
Electrode x Expression: i 7 ( 6; 13 ) = 0.76, p = 0.617, r\p = 0.259; 
Electrode x Expression x Condition: F(e, 13) = 1.67, p = 0.206, 
T]2 = 0.435]. 

Because Expression showed no effect, subsequent t-tests with 
the mean amplitude values averaged over responses to fearful 
and happy faces were applied separately to data measured from 
each electrode in order to compare the responses between the 
stimulus presentation conditions. The stimulus presentation con- 
dition had a significant effect on differential responses (emotional 
minus neutral) at P7, t(\$) = 3.38, p = 0.011 (mean difference 
0.31|xV, 95% confidence interval 0.12-0.51 |xV) and at Pz elec- 
trodes, f ( i8) = 2.13, p = 0.047 (mean difference 0.21|xV, 95% 
confidence interval 0.003-0.42 |xV). There was also a marginally 
significant effect at Oz electrode, t(\$) = 2.07, p = 0.053 (mean 
difference 0.27 |xV, 95% confidence interval 0.04-0.55 |xV). For 



all these electrodes (P7, Pz, Oz), the difference- wave amplitudes 
were larger (more negative) in the oddball condition than in 
the equiprobable condition (Figure 7). For the other electrodes, 
no significant differences between the conditions were observed. 
Amplitude values differed from zero at all electrode sites and 
in both stimulus presentation conditions. Table 1 shows the t- 
values, p-values, mean differences, and 95% confidence intervals 
for the differential response amplitude values at each electrode 
tested against zero (one sample f-test). 

Since visual inspection of the scalp topographies suggested that 
the lateral parietal activity was right dominant in the equiproba- 
ble condition, while no such lateralization existed in the oddball 
condition, Electrode x Condition effect was further studied. 
Pair-wise comparisons were applied for the amplitude values 
recorded at P7 and P8 separately for the conditions. The statistics 
conformed to visual observation showing that, in the equiprob- 
able condition, amplitude values were larger in the right pari- 
etal electrode site than the left (i.e., P8 vs. P7), tp) = 3.19, 
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FIGURE 5 | Back-projected 170-ms components for each participant. 

Differential response waveforms (emotional minus neutral) at P7 and P8 
electrodes are drawn separately for fearful and happy faces and for oddball 
condition and equiprobable condition groups. 



p = 0.022 (Bonferroni corrected, mean difference 0.62 |xV, 95% 
confidence interval 0.18-1.06 |xV). In the oddball condition, no 
such difference was found, t@) = 0.48, p = 0.641 (mean differ- 
ence 0.05 |xV). Figure 8A depicts lateralization index for both 
conditions (oddball and equiprobable). Comparison of the lat- 
eralization indexes in the oddball and equiprobable conditions 
indicated a significant difference between the conditions, t(\$) = 
2.36, p = 0.030, mean difference 0.40 |xV, 95% confidence inter- 
val 0.04-0.75 |xV). 

170-MS COMPONENT 

Figure 9 shows the scalp potential maps for the back-projected 
170-ms component. A MANOVA indicated a main effect 
for electrode, P (6 , 13) = 14.11, p < 0.0001, ^ = 0.867. No 
other main effects or any of the interaction effects were 
significant (Expression: F(\ t is) = 0.35, p = 0.563, r\p = 0.019; 
Electrode x Condition: F (6j 13) = 0.68, p = 0.669, \]j = 0.239; 



130 ms 

EQUIPROBABLE ODDBALL 





FIGURE 6 | Scalp potential maps of the 130-ms component 
back-projected to the electrodes. Map for the equiprobable condition 
group on left and map for the oddball condition group on right. 
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Electrodes 

FIGURE 7 | Mean amplitude values (|xV), confidence intervals, and 
scatterplots of the individual participants' values for each electrode in 
the equiprobable condition (EQ) and oddball condition (OB) for the 
130-ms component (differential response; emotional minus neutral 
face). An asterisk (*) indicates a significant difference (p < 0.05) between 
conditions at P7 and Pz electrodes. 



Expression x Condition: P(i,i8) = 0-66, p = 0.800, r\j = 0.004; 
Electrode x Expression: F($ t 13) = 0.56, p = 0.752, r\p = 0.207; 
Electrode x Expression x Condition: i 7 ^ 13) = 0.34, p = 0.901, 
r\p = 0.137). The effect for electrode resulted from the ampli- 
tudes being of positive polarity in the anterior electrodes and 
of negative polarity in the posterior electrodes (Figures 9, 10). 
Amplitude values averaged for the fearful and happy expressions 
differed from zero at all the electrode sites and in both stimu- 
lus presentation conditions. Table 2 shows the f-values, p-values, 
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Table 1 | 130-ms component. 



Electrode 


f 


n 

H 


Mean 


95% Confidence 


site/ 




(2-tailed) 


difference 


interval of the 


condition 






(jjlV) 


difference 










Lower 


Upper 


Fz/OB 


5.98 


U.UUU1 


0.68 


0.42 


0.94 


Fz/EQ 


6.24 


U.UUU1 


0.72 


0.46 


0.99 


F3/OB 


5.40 


U.UUUI 


0.72 


0.42 


1.03 


F3/EQ 


4.73 


0.001 


0.64 


0.33 


0.94 


F4/OB 


6.78 


U.UUUI 


0.66 


0.44 


0.88 


C/1 /CO 

r4/bU 


5.71 


U.UUU I 


0.60 


0.36 


0.84 


rz/Ub 


—6.12 


U.UUU I 


—0.53 


-0.73 


-0.34 


Pz/EQ 


-6.54 


0.0001 


-0.32 


-0.43 


-0.21 


P7/OB 


-11.08 


0.0001 


-0.65 


-0.78 


-0.52 


P7/EQ 


-4.75 


0.001 


-0.34 


-0.50 


-0.18 


P8/OB 


-7.38 


0.0001 


-0.70 


-0.92 


-0.49 


P8/EQ 


-5.08 


0.001 


-0.96 


-1.38 


-0.53 


Oz/OB 


-7.68 


0.0001 


-0.81 


-1.05 


-0.57 


Oz/EQ 


-6.65 


0.0001 


-0.53 


-0.72 


-0.35 



One sample t-tests (tested against 0) for the emotional minus neutral differential 
responses. OB, oddball condition; EO, equiprobable condition. 



Lateralization index 
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FIGURE 8 | Lateralization index for the 130-ms (A) and 170-ms (B) 
components separately for both conditions (oddball = OB and 
equiprobable = EQ). The values are calculated for the back-projected 
components' amplitudes from electrodes P7 (left) and P8 (right) as follows: 
(left - right)/(left + right). The bars represent the mean values in the group 
and the whiskers 95% confidence intervals. Individual participants' values 
are marked with filled circles. 



mean differences, and 95% confidence intervals for the differen- 
tial response amplitude values at each electrode tested against zero 
(one sample f-test). 

Figure 8B shows lateralization index for both conditions (odd- 
ball and equiprobable). No statistically significant difference was 
found in the lateralization indexes between the conditions, t(\$) = 
1.06, p = 0.304. 

DISCUSSION 

We presented two groups of adults with a series of pictures 
of faces: for one group the faces were presented in an oddball 
condition, for the other group the faces were presented in an 
equiprobable condition. Facial identities changed on a trial-by- 
trial basis. For the oddball condition group, most of the faces 
expressed neutral emotion, with rare happy and fearful faces 



170 ms 

EQUIPROBABLE ODDBALL 





FIGURE 9 | Scalp potential maps of the 170-ms component 
back-projected to the electrodes. Map for the equiprobable condition 
group on left and map for the oddball condition group on right. 
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FIGURE 10 | Mean amplitude values (11 V) and confidence intervals, and 
scatterplots of the individual participants' values for each electrode in 
the equiprobable condition (EQ) and oddball condition (OB) for the 
170-ms component (differential response; emotional minus neutral 
face). No significant differences between conditions were found. 



randomly violating this regularity For the equiprobable con- 
dition group, neutral, happy, and fearful faces were presented 
with equal probability and formed no regularity in the stimulus 
series. 

Differential responses to emotional expressions (fearful- 
neutral and happy-neutral) were calculated, wavelet filtering was 
applied to the averaged data in order to increase the signal-to- 
noise ratio and the independent components were extracted by 
the open-source ICA software, ICASSO (Himberg et al., 2004). 
We found two separate components for the emotional faces in 
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Table 2 | 170-ms component. 


Electrode T p 


Mean 


95% Confidence 


site/ (2-tailed) 


difference 


interval of the 


condition 


(*iV) 


difference 






Lower Upper 



0.89 
0.82 
0.83 
0.63 
0.82 
0.62 
0.34 
0.30 
0.24 
0.25 
0.70 
0.34 
0.24 
0.23 

One sample t-tests (tested against 0) for the emotional minus neutral differential 
responses. OB, oddball condition, EQ, equiprobable condition. 

both the oddball and equiprobable conditions: one at the latency 
of ~ 1 30 ms and the other at the latency of ~ 1 70 ms after stimulus 
onset. 

The 170-ms component conforms to the face -sensitive N170 
response. The scalp topography of the component extracted from 
the differential response included a lateral occipito-parietal nega- 
tivity and frontal positivity, and was thus similar to the topogra- 
phy previously reported for N170 itself (e.g., Bentin et al, 1996; 
Ashley et al, 2004; Williams et al, 2006; Blau et al, 2007). Also 
Blau et al. (2007) have found that subtracting responses to task- 
irrelevant fearful faces from those to neutral faces provided a 
topography that was highly similar to the topography of N 170. 
The frontal positivity was most likely a so called vertex positive 
potential (VPP) known to be elicited by the same brain gener- 
ators as N170 (Joyce and Rossion, 2005). Also the present data 
analysis based on ICA method supports this view. 

The 170-ms component did not, as expected, differ between 
the stimulus presentation conditions, but differentiated between 
the emotional and neutral faces. Emotional modulation of N 170 
has also been reported in several other studies (Batty and Taylor, 
2003; Eger et al, 2003; Caharel et al, 2005; Williams et al, 2006; 
Blau et al., 2007; Leppanen et al., 2007; Schyns et al., 2007; Japee 
et al., 2009; Vlamings et al., 2009; Wronka and Walentowska, 
2011). Since there are also studies in which no N 170 modulation 
for emotional expressions has been found (Eimer and Holmes, 
2002; Eimer et al, 2003; Holmes et al, 2003, 2005; Ashley et al, 
2004; Susac et al, 2004; Santesso et al, 2008), future studies 
should explore the factors influencing this modulation. These 
could be related to several methodological choices, for example, 
to the behavioral tasks the participants are asked to perform dur- 
ing stimulus presentation. Recently, the location of the reference 
electrode has also been suggested to have an effect on the N170 
modulation by emotional expression (Rellecke et al., 2013). 



The finding of a 130-ms component in both conditions was 
unexpected. It was observed as enhanced parieto- occipital nega- 
tivity and frontal positivity to the emotional faces in comparison 
to neutral faces in both conditions. Importantly, however, dif- 
ferences in topography between the conditions were observed: 
the topography was bilateral over the lateral parietal sites in 
the oddball condition while it was more right- dominant in the 
equiprobable condition. The bilateral posterior topography of the 
component conforms to vMMN to facial expressions (Astikainen 
and Hietanen, 2009; Stefanics et al, 2012). Also, the observed 
frontal positivity in the present data replicates our previous find- 
ings of the vMMN topography to emotional faces (Astikainen 
and Hietanen, 2009). The current data suggest that this differ- 
ential response is not due to detection of the regularity violation, 
but more generally related to emotional processing since it was 
elicited also in the equiprobable condition. Indeed, in some pre- 
vious studies investigating ERPs to facial expressions, but not 
applying the oddball condition, a frontal positivity to emotional 
expressions relative to neutral ones at a latency corresponding to 
that observed in the present study has been found (Eimer and 
Holmes, 2002; Kiss and Eimer, 2008). The latency of the 130- 
ms component is in line with the earliest differential responses to 
changes in facial expressions (Susac et al., 2004, 2010; Astikainen 
and Hietanen, 2009; Chang et al, 2010; Stefanics et al, 2012). The 
fact that elicitation of the 130-ms component was also observed 
in the equiprobable condition suggests that it reflects both the 
detection of the regularity violations and encoding of the emo- 
tional information in the faces. This finding calls for appropriate 
control conditions, such as an equiprobable condition, in future 
studies of vMMN to facial expressions. 

In accordance with our previous finding of vMMN in a simi- 
lar experimental paradigm as the oddball condition applied here 
(Astikainen and Hietanen, 2009), the 130-ms component showed 
no difference between fearful and happy expressions in either 
condition (oddball or equiprobable). Also in a study in which 
schematic faces were applied, sad and happy faces as rare changes 
among neutral standard faces elicited equally large amplitudes 
(Chang et al., 2010). On the other hand, a so called "negativ- 
ity bias" in response latencies has been reported in two previous 
vMMN studies using facial expressions (Kimura et al, 2011; 
Stefanics et al., 2012). In these studies, fearful faces as deviants 
elicited differential responses in clearly earlier latency ranges than 
happy faces; for example, the differential response found in the 
70-120-ms latency range to fearful faces was absent for happy 
faces (Stefanics et al., 2012). In the present study, the fearful 
and happy deviants elicited the same components (130— and 
170-ms component) and they were also similar in their laten- 
cies. The lack of a negative bias in our study is not, however, 
in conflict with the previous findings by Kimura et al. (2011) 
and Stefanics et al. (2012), hence we only analyzed the com- 
ponents in the relevant latency range for N170 (100-200 ms 
post stimulus). Inspection of the components extracted by ICA 
for the entire post stimulus time period might have revealed 
emotion-specific components in the earlier and later latency 
ranges as well. 

There are some limitations in the present study. First, the emo- 
tional expressions were not presented with the same probability 
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in the oddball and equip robable condition (p = 0.1 and p = 
0.33, respectively). It is thus possible that it was not solely 
the differences in cognitive expectation (present only in the 
oddball condition in which the high probability of the neu- 
tral standard faces formed it), but also the differences in the 
probability of the emotional expressions as such that could 
have induced the between- condition effects. In the future, one 
should investigate whether the response amplitude of vMMN 
or N170 to emotional faces is influenced by the presenta- 
tion probability within the stimulus sequence. Second, the 
study was conducted with a limited sample size. Future stud- 
ies should aim to replicate the findings with a larger num- 
ber of participants. Third, the current study is based on EEG 
data recorded with a montage of 14 electrodes. More sensors 
could have allowed for example estimation of the locations of 
sources (for a magnetoencephalography study of facial process- 
ing see, Smith et al, 2009). Finally, the present study does not 
reveal to which specific diagnostic features in faces the found 
components are responses to (see e.g., Schyns et al., 2009). 
However, in the present study, several different facial identi- 
ties in the pictures were applied and there were no immediate 
repetitions in them. Our results might thus reflect abstraction 
of emotion-related features among several changing low-level 
features. 



In sum, we found two separate components in the 100-200-ms 
latency range for changes in emotional expressions. The com- 
ponent peaking at ~ 170 ms post stimulus showed no difference 
between the stimulus presentation conditions and it was identi- 
fied as the face -sensitive N170 response. A component peaking at 
130 ms post stimulus was different in its scalp topography in the 
oddball and the equiprobable conditions, i.e., when the presented 
face violates the regularity formed by the standard faces in com- 
parison to the condition in which no regularity is present. Future 
studies of vMMN to facial expressions should apply relevant con- 
trol conditions to avoid the confounding effect of the encoding of 
emotional expressions as such. 
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SUPPLEMENTARY MATERIALS 

A data sample and Matlab scripts for running the data pro- 
cessing including wavelet filtering and ICA can be found 
online at: http://www.frontiersin.org/Human_Neuroscience/ 
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